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1.   Introduction 

Research  on  implementation  in  the  past  decade  has  been  guided 
almost  exclusively  by  a  single  paradigm,  the  factor  model.  This  re- 
search generally  starts  with  a  group  of  variables  or  sets  of  variables 
potentially  relevant  to  implementation  outcomes,  and  attempts  to  de- 
termine the  relative  importance  of  these  variables  or  sets  of  variables 
to  these  outcomes  (we  use  the  term  factor  to  refer  to  either  a  single 
variable  or  a  cluster  of  variables,  as  both  tacks  can  found  in  exist- 
ing factor  research).  The  basis  for  deciding  which  variables  to  in- 
clude in  a  factor  study  is  complex,  as  there  are  a  large  number  of  can- 
didates. Two  major  approaches  to  the  selection  of  factors  for  studying 
can  be  found  in  the  literature:  the  inductive  approach,  which  starts 
with  a  source  of  data  (e.g.,  ease  observations)  and  extracts  those 
factors  appearing  to  be  important;  and,  the  deductive  approach,  which 
selects  factors  for  study  before  data  collection  on  the  basis  of  past 
research,  literature  review,  expert  opinion,  etc.  Certain  other  varia- 
tions in  research  approach  can  be  observed  within  the  factor  paradigm 
--  focusing  on  one  particular  type  of  factor  vs.  looking  at  a  range 
of  factors,  examining  a  single  project  in  depth  vs.  surveying  multiple 
projects  and  organizations  --  but  in  all  cases  the  underlying  question 
is,  what  factors  are  associated  with  implementation  success. 

Given  the  quantity  of  effort  put  into  this  single  research 
paradigm,  we  could  expect  to  find  some  definitive  answers  to  the 
implementation  problem  emerging.  And,  a  glance  at  the  literature 
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would  suggest  that  this  might  be  the  case;  we  all  know  that  top  manage- 
ment support  and  user  involvement  are  critical  to  successful  implemen- 
tations. But,  beyond  this,  what  do  we  know  about  implementation;  and, 
how  sure  are  we  that  this  type  of  management  support  and  user  involve- 
ment 2S_  important?  Or,  more  critically,  when  are  these  factors 
important? 

In  this  paper,  we  will  review  a  number  of  these  factor  re- 
search efforts.  We  will  consider  the  studies  both  individually  and  as 
a  group,  attempting  to  find  the  general  principles  which  have  emerged 
from  this  research  paradigm.  On  the  basis  of  our  findings  in  review- 
ing this  research,  we  will  draw  some  conclusions  about  the  most  profita- 
ble avenues  for  further  research  on  implementation. 

2.  Research  Efforts  Reviewed 
2.1.  Bases  for  selection. 

The  relatively  small  amount  of  research  which  has  been  con- 
ducted on  implementation  precluded  the  need  for  elaborate  sampling 
procedures  in  selecting  the  studies  to  be  considered  in  this  review 
effort.  Clearly,  the  most  important  factor  for  a  study's  inclusion 
was  my  knowing  (or  finding  out)  that  it  existed.  While  this  may  not 
seem  to  be  the  most  scientific  basis  for  selection,  the  studies  seem 
to  reflect  the  full  i^ange  of  research  on  OR/MS  activities.  They  can 
be  classified  in  terms  of  three  important  characterisitcs:  1)  the 
settings  included  in  these  studies,  2)  the  measurement  methodologies 


employed,  and  3)  the  dependent  variables  chosen.   In  considering  these 
characteristics  we  should  keep  two  questions  in  mind:  are  the  studies 
included  here  representative  of  factor  research  on  implementation?  and, 
are  they  representative  of  research  on  implementation  generally?  Turn- 
ing first  to  the  settings  included  in  the  studies,  we  find  a  wide  range 
of  coverage  --  a  substantial  number  of  public  and  private  organizations, 
a  variety  of  organization  sizes,  a  number  of  cultures  (U.S.,  Europe, 
and  Latin  America),  and  a  variety  of  project  types.  While  no  single 
study  covers  the  entire  range,  taken  as  a  whole  they  seem  to  provide 
an  adequate  representation  of  the  non-military  settings  in  which  OR/MS 
activity  is  taking  place. 

In  considering  the  measurement  methodology  employed,  two  issues 
are  of  relevance.  First,  what  variables  were  selected  for  measurement 
and  what  was  the  basis  for  their  selection?  And,  second,  how  were  they 
actually  measured?  Keen  (1974)  suggests  that  factor  studies  typically 
measure  those  variables  which  are   most  readily  measurable,  with  little 
regard  for  theoretical  considerations  in  the  selection  of  these  vari- 
ables. The  fourteen  studies  included  in  our  sample  identify  approxi- 
mately 140  different  factors,  covering  a  wide  range  of  the  variables 
potentially  affecting  implementation  outcomes.  We  shall  show  later, 
however,  that  the  distribution  of  research  effort  among  these  factors 
is  far  from  balanced.  The  bases  on  which  these  factors  were  selected 
for  study  is  not  immediately  apparent;  few  authors  state  explicitly  why 
they  have  chosen  the  factors  they  did,  and  in  at  least  six  of  the 
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fourteen  studies  the  factors  examined  were  abstracted  from  the  data 
(normally  interview  or  case  observation  based)  after  it  was  collected. 
We  will  return  to  this  question  in  a  later  section,  but  for  now  we  can 
state  that  there  is  no  immediate  grounds  for  dismissing  Keen's  con- 
jecture. Turning  to  the  question  of  measurement  itself  we  find  a 
variety  of  techniques  employed,  ranging  from  introspection,  through 
case  observation,  to  administration  of  questionnaires  and  the  examina- 
tion of  company  records.  Thus,  on  the  issue  of  the  completeness  of 
coverage  of  measurement  methodology  in  our  sample  of  studies  we  can 
conclude  that  the  range  has  been  adequately  covered. 

The  third  characteristic  to  be  considered  is  that  of  the  de- 
pendent variable  selected  and  the  basis  for  its  measurement.  The  four- 
teen studies  show  considerable  variety  on  this  dimension,  including 
both  "hard"  --  e.g.,  percent  of  projects  implemented,  specific  types 
of  system  usage  --  and  "soft  --  e.g.,  perceived  project  usefulness, 
rank  ordering  of  perceived  achievement  in  computer  use  --  measures 
of  outcome.  We  will  review  these  measures  in  more  detail  when  we  look 
at  the  individual  studies. 

In  general,  we  can  assert  that  the  sample  of  studies  con- 
sidered displays  a  considerable  range  on  each  of  these  three  critical 
dimensions,  and  that  it  is  representative  of  the  bulk  of  research  on 
MS/OR  and  computer  application  implementation.  Certain  studies  which 
could  have  been  included  in  this  sample  were  purposely  excluded  because 
they  focused  on  a  single,  specific  factor  rather  than  on  the  more 


general  question  of  the  determinants  of  implementation  outcomes.  The 
work  of  Huysmans  (1970)  or  Doktor  and  Hamilton  (1973)  on  the  importance 
of  cognitive  style  falls  into  this  category,  and  we  do  not  mean  to  deny 
the  relevance  of  this  work.  It  does,  however,  represent  only  one  among 
the  many  issues  potentially  relevant  to  the  implementation  process,  and 
does  not  focus  on  the  central  question  of  the  factor  study:  "What 
variables  are  most  relevant  to  implementation?" 

The  next  section  reviews  briefly  the  fourteen  studies  we 
have  included  in  our  sample.  We  will  look  at  the  issues  of  setting 
and  methodology,  plus  certain  key  points  brought  out  by  these  studies. 
Consideration  of  actual  results  will  be  left  to  a  later  section. 

2.2.  Once  over  lightly:   highlights  of  the  fourteen  studies. 

1.  McKinsey  &  Co.  (1968):   "Unlocking  the  Computer's  Profit 
Potential" 

While  actually  conducted  later  than  some  of  the  other  studies 
in  the  sample,  this  work  is  conceptually  first,  as  it  is  largely  an 
update  of  an  earlier  McKinsey  study  (Garrity,  1963).  The  focus  of  this 
study  was  on  those  factors  which  differentiated  more  successful 
computer  users  from  their  less  successful  counterparts.  The  sample 
included  36  companies  in  13  industries,  covering  a  wide  range  of  sizes, 
and  located  both  in  the  U.S.  and  in  Europe.  Success  was  measured  by 
judgemental ly  ranking  the  36  companies  on  their  overall  achievement 
with  computers,  and  then  comparing  the  first  18  companies  with  the  last 


18.  Foctors  considered  in  this  study  were  derived  inductively,  on 
the  basis  of  their  apparent  importance  to  the  organization's  computer 
success,  from  extensive  interviews  with  both  staff  and  line  executives. 
The  McKinsey  studies  are  notable  both  for  being  early  in  the  history 
of  studying  implementation  and  for  their  best  known  results,  the 
importance  of  top  management  support  of  and  operating  management 
involvement  in  the  application  development  process. 

2,  Evan  &  Black  (1967):   "Innovation  in  Business  Organiza- 
tions: Some  Factors  Associated  with  Success  or  Failure 

of  Stoff  Proposals" 
Evan  and  Black  focus  on  innovation,  "the  implementation  of 

new  procedures  or  ideas"  (p.  519),  and  attempt  to  determine  what 
factors  affect  the  likelihood  that  an  innovative  proposal  submitted  by 
staff  to  line  management  will  be  implemented.  The  sample  considered 
in  this  study  included  52  respondents  in  a  smaller  number  of  companies. 
The  respondents  were  either  staff  personnel  or  line  managers,  and 
each  reported  on  one  successful  and  one  unsuccessful  proposal  for 
innovation  (success  being  defined  as  the  proposal's  being  "largely 
accepted  and  implemented  by  management").  Data  was  collected  with 
self-reporting  questionnaires  asking  closed-ended  questions  about 
eleven  pre-selected  factors.  The  data  was  then  analyzed  using  stepwise 
discriminant  analysis  to  select  those  factors  (from  among  the  eleven) 
which  best  separated  successful  from  unsuccessful  proposals.  Seven 
of  the  eleven  factors  were  thus  found  to  be  significant  at  the  .05 


level,  though  the  authors  claim  to  see  no  a  priori  reason  why  these 
should  be  any  more  important  than  the  other  four. 

3.  Rubenstein  et  al .  (1967):   "Some  Organizational  Factors 
Related  to  the  Effectiveness  of  Management  Science  Groups 
in  Industry" 

The  third  study  considered  is  reported  in  Rubenstein  et  al. 
(1967).  While  we  have  included  only  the  results  discussed  in  this 
single  paper,  they  are  representative  of  other  work  reported  by  these 
authors  (e.g.,  Radnor  et  al.,  1968  &  1970).  This  work  is  concerned 
with  finding  the  "necessary  and  sufficient  conditions  which  permit  the 
integration,  acceptance,  and  growth  of  [innovative]  activities",  partic- 
ularly OR/MS,  in  organizations;  that  is,  the  factors  contributing  to 
greater  use  of  OR/MS.  To  operational ize  this  concept  the  authors 
examine  the  extent  to  which  proposals  and  programs  of  the  organization's 
OR/MS  activity  were  being  implemented.  As  no  hard  measure  is  specified, 
we  assume  that  it  is  the  researcher's  impression  of  how  well  these 
processes  were  going  that  serves  as  the  dependent  variable.  The 
factors  considered  in  this  study  were  those  appearing  important  to  the 
authors  after  some  preliminary  exploration.  No  mention  is  made  of  how 
variables  to  be  considered  in  these  preliminary  studies  were  selected. 


* 
We  offer  the  suggestion  that  some  of  these  factors  were  not  adequately 

tested;  the  specific  items  in  the  questionnaire  well  may  not  be  measur- 
ing what  they  purport  to  be. 


r.ictnrs  were  assessed  throuqfi  interviews  of  personnel  involved  in  the 
cases  studied.   It  is  interesting  to  note  that  Rubenstein  et  al.  are 
among  the  first  researchers  to  suggest  the  potential  importance  of 
contingencies;  in  particular,  they  argue  for  the  importance  of  the 
organization's  stage  in  the  life  cycle  of  OR/MS  activities  as  a 
conditioning  variable. 

4.  Harvey  (1970):   "Factors  Making  for  Implementation 
Success  and  Failure" 

Harvey's  study  differs  from  the  others  in  our  sample  in  a 
number  of  respects.  First,  it  is  not  truly  a  research  effort,  but  more 
a  case  of  group  introspection  to  explain  past  successes  and  failures. 
Second,  and  more  important  from  our  point  of  view,  it  occurred  because 
the  author  felt  that  the  literature  tended  to  rely  too  much  on  a 
single  explanation  --  management  support  and  participation  --  for  the 
success  or  failure  of  OR/MS  implementations,  and  that  other  factors 
of  at  least  equal  importance  should  be  explored.  Thus,  the  sample 
analyzed  in  this  study  consisted  of  31  former  clients  of  the  author's 
consulting  firm;  this,  he  claims,  may  have  biasod  the  results,  as  all 
of  these  companies  "were  looking  for  state-of-the-art  solutions  to 
problems  ..."  (p.  B313).  The  factors  considered  were  those  that  the 
author  and  his  colleagues  felt,  based  on  their  experience,  were  import- 
ant. Each  of  the  31  companies  in  the  sample  was  ranked  by  the 
consultants  on  each  factor.  The  dependent  variable  was  simple:  had 
the  company  accepted  the  consultants'  recommendations  and  implemented 
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them.  As  the  eleven  factors  selected  for  testing  were  in  a  sense 
derived  from  the  data  used  to  test  them,  it  is  not  surprising  to  find 
that  all  eleven  were  found  to  correlate  highly  with  implementation 
success.  We  also  should  note  that  the  factors  which  Harvey  initially 
argued  should  not  be  the  sole  explanations  of  success  --  i.e.,  support 
and  involvement  --  are  not  considered  at  all  in  his  study. 

5.  Dickson  A  Powers  (1971):   "MIS  Project  Management:  Myths, 
Opinions,  and  Reality" 

Dickson  and  Powers  focused  their  attention  on  the  "organiza- 
tional and  procedural  factors"  correlating  with  success  in  a  certain 
class  of  MIS  projects.  The  study  included  two  projects  in  each  of  ten 
organizations  in  the  flinneapol is/St .  Paul  area.     The  factors  studied 
were  selected  in  a  relatively  complex  fashion.  First,  a  moderately 
long  list  of  potential  factors  (approximately  35)  was  generated  by  the 
authors  and  their  students.  This  list  was  then  submitted  to  a  large 
panel  of  DP  professionals  (SMIS  members)  for  their  ratings  of  factor 
importance.  Their  responses  (approx.  140  returned  questionnaires)  were 
used  to  compute  average  factor  'importance  scores'  and  to  develop 
clusters  of  factors  (using  factor  analysis).  A  subset  of  these  items, 
spanning  the  range  of  the  a  priori  'importance  scores'  and  covering 
all  factor  clusters,  was  selected  for  use  in  tin's  study.  Additional 
items  were  included  where  data  collection  would  be  easy.  Collecting 
data  for  the  study  was  accomplished  by  extensive  interviewing  of 
management  and  technical  personnel  in  each  of  the  ten  firms.  Four 
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separate  indicators  of  success  were  employed  as  dependent  variables  -- 
actual  vs.  estimated  development  time,  actual  development  cost  vs. 
budget,  user  satisfaction,  and  the  impact  of  the  project  on  computer 
operations.  One  of  the  more  interesting  findings  of  this  study  was 
the  total  lack  of  correlation  among  these  four  measures. 

6.  Vertinsky  (1972):   "OR/MS  Implementation  in  Valle, 
Colombia,  S.A.:  A  Profile  of  a  Developing  Region" 

This  study  widens  considerably  the  coverage  of  settings  in 
our  sample,  as  it  looks  at  OR/MS  activities  in  Colombia,  South  America. 
The  author  is  concerned  chiefly  with  the  correlates  of  implementation 
and  diffusion  of  OR/MS  in  developing  regions.  Selection  of  factors 
for  this  study  was  based  both  on  induction  (those  factors  appearing 
important  in  an  earlier  study)  and  on  deduction  (factors  which  theoret- 
ically should  be  important  in  a  developing  country).  Factor  measure- 
ment was  accomplished  through  a  mixture  of  interviews  and  questionnaires, 
the  respondents  being  Presidents  or  General  Managers  of  the  companies 
under  study.  The  dependent  variable  in  this  study  appears  to  be  the 
researcher's  assessment  of  whether  the  factor  impacted  the  OR/MS 
diffusion  and  implementation  processes. 

7.  Drake  (1973):  The  Administration  of  Transportation 
Modeling  Projects 

Drake's  work  focuses  on  a  specific  type  of  project,  the 
application  of  OR/MS  to  transportation  system  modeling  efforts.  He 
surveyed  a  large  number  of  such  projects  (approx.  50,  with  25  in 
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considerable  detail)  based  in  both  the  U.S.  and  Europe,  and  having  both 
private  and  governmental  clients.  The  factors  considered  were  designed 
to  test  a  series  of  hypotheses  proposed  by  the  author  (the  basis  for 
selecting  these  particular  hypotheses  appears  to  be  Drake's  own  work 
experience).  Data  were  collected  through  a  detailed  questionnaire 
administered  by  the  researcher,  and  this  data  was  supplemented  by  his 
field  observations.  Drake's  intention  was  to  obtain  responses  from  both 
the  model  builder  and  the  decision  maker  (model  user),  though  in  a 
number  of  cases  he  was  able  to  speak  only  to  the  user.  The  dependent 
variable  chosen  was  the  decision  maker's  assessment  of  the  project's 
usefulness.  This  focus  on  the  user's  view  of  an  implementation  effort 
is  both  rare   and  quite  important;  much  of  the  literature  is  based  on 
the  technician's  view  of  the  project,  which  can  be  quite  different  from 
that  of  the  user  (see  Dickson  &  Powers  for  a  discussion  of  designer/ 
user  differences  in  perception). 

8.  Lucas  (1973):   "Behavioral  Factors  in  System  Implementa- 
tion" 

Lucas'  study  differs  from  most  of  the  others  we  will  consider. 
He  looks  at  a  single  sales  information  system  having  numerous  potential 
users  (salesmen)  in  a  single  company,  and  attempts  to  find  those  factors 
which  help  differentiate  users  from  non-users.  Lucas  makes  the  argu- 
ment that  for  purposes  of  implementation  a  M.I.S.  is  equivalent  to  any 
other  MS/OR  project;  at  a  minimum,  an  information  system  is  a  simple 
model,  and  frequently  information  systems  contain  relatively 
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sophisticated  models.  This  arqument,  which  wo  largely  accept,  should 
explain  our  rationale  for  includinq  this  (as  well  as  some  other)  studies 
in  our  sample.  Lucas  presents  a  simple  descriptive  model  of  factor 
clusters  purported  to  explain  system  use,  and  from  this  model  derives 
a  number  of  factors  to  be  tested.  Factors  as  well  as  dependent 
variables  were  measured  with  questionnaires  and  by  examining  company 
records.  Indices  of  six  specific  types  of  system  usage  served  as  the 
dependent  variables,  and  factor  importance  was  assessed  using  stepwise 
multiple  regression  to  develop  usage  "prediction"  equations.  All  factors 
were  found  to  be  significant  in  at  least  one  such  equation,  but  the 
particular  factors  included  varied  across  equations;  those  dealing  with 
attitudes  towards  and  perceptions  of  the  computer  system  showed  a  rela- 
tively consistent  positive  relationship  with  usage,  with  the  results  for 
environmental  and  personal  factors  being  considerably  weaker  and  less 
consistent. 

9.  Manley  (1971):  "Implementation  Attitudes:  A  Model  and  a 
Measurement  Methodology" 

Manley' s  work  differs  in  a  number  of  respects  from  all  of  the 
other  studies  in  our  sample.  First,  it  focuses  neither  on  the  model 
builder  nor  the  model  use'-,  but  on  the  "client",  defined  as  a  person 
impacted  by  the  proposed  innovation  but  having  no  de  jure  authority  to 
decide  whether  to  accept  .md  use  it  or  not.  The  clients  in  this  case  were 
teachers  in  a  public  school  system  presented  with  a  proposal  for 
"optimizing  the  computation  of  the  supplemental  pay  for  extra-curricular 
activities."  The  second  iiajor  difference  was  in  the  dependent  variable. 
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an  index  predicting  the  level  of  client  group  support  of  or  resistance 
to  the  proposed  project.  This  index  was  derived  from  the  interaction  of 
the  clients'  assessment  of  the  relative  importance  of  five  external 
factors  and  their  "attitude  orientations"  toward  these  factors.  The 
external  factors  considered  were  arrived  at  by  boiling  down  a  list  of 
factors  culled  from  the  literature  to  five  generic  factors.  The 
measurement  of  these  factors  represents  the  third,  and  probably  the  most, 
unique  aspect  of  this  study.  While  being  conducted  in  a  real  organiza- 
tion with  respondents  who  believed  the  proposal  to  be  genuine,  the  study 
was  in  fact  a  controlled  experiment;  factors  were  varied  systematically 
by  giving  different  respondents  different  versions  of  the  proposal.  In 
the  experiment  Manley  tested  two  of  the  five  external  factors  --  manage- 
ment support  for  the  project  and  the  amount  of  the  client's  time 
required  for  implementation  --  and  found  both  to  result  in  significant 
differences  in  the  client's  "attitude  orientations". 

10.  Gibson  (1973):  "A  Methodology  for  Implementation  Research" 
Gibson  offers  the  only  example  of  a  single  case  study  that  we 
have  included  in  our  sample.  His  research  was  conducted  in  a  New 
England  bank,  and  was  concerned  with  an  effort  to  implement  a  translator 
model  which  would  interface  with  a  regional  economic  model  being  developed 
concurrently.  The  factors  considered  in  this  research  were  arrived  at 
inductively,  being  those  issues  which  seemed  to  have  important  impacts 
on  the  progress  of  this  implementation.  Thus,  selection  of  factors  was 
tied  directly  to  the  dependent  variable.  We  should  note,  however,  that 
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there  was  a  fairly  long  history  of  "action  research"  by  Gibson  and  his 
colleagues  at  this  site,  adding  credebility  to  the  importance  of  these 
factors  in  this  setting.  "Measuring"  factors  was  accomplished  primarily 
through  unstructured,  periodic  interviews  with  key  personnel,  but 
supplemented  with  questionnaires  to  explore  certain  issues  which  arose. 
This  is  the  only  study  in  our  sample  where  the  researchers  acted  as 
participant  observers  and  actually  attempted  to  influence  the  implement- 
ation process. 

Including  Gibson's  study  in  our  review  of  factor  research  might 
seem  inappropriate,  but  we  have  done  so  for  two  reasons  --  first,  the 
results  of  the  study  are  couched  to  an  extent  in  terms  of  factors;  and, 
his  work  presents  one  extreme  in  methodology  for  studying  implementation, 
thus  adding  to  the  range  of  our  sample.  In  fairness  to  the  reader  we 
must  admit  that  this  is  not  really  a  factor  study.  And,  in  fairness  to 
Gibson  we  must  point  out  that  this  work  does  not  suffer  from  a  number  of 
methodological  problems  which  plague  factor  research  and  which  we  will 
discuss  later  in  this  paper.  Indeed,  Gibson  is  one  of  a  very  few 
researchers  in  this  area  sensitive  to  methodological  issues. 

n.  Smith  et  al .  (1973):  "Operations  Research  Effectiveness: 
An  Empirical  Study  of  Fourteen  Project  Groups" 

Our  next  study  also  took  place  in  a  single  organization,  but  it 
looked  at  fourteen  separate  projects  within  this  organization.  Smith  et 
al .  attempted  to  find  the  factors  which  contributed  to  the  effectiveness 
of  project  teams  consisting  of  user,  systems,  and  OR  personnel.  The 
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factors  considered  included  a  number  of  predefined  items  --  e.g.,  user 
demographics  --  plus  those  factors  reported  by  team  members  to  have  been 
important  contributors  to  or  detractors  from  their  effort.  Factors  were 
measured  using  a  questionnaire  administered  by  the  researchers.  Two 
dependent  variables  were  employed.  For  testing  the  predefined  factors, 
the  average  group  effectiveness  rating  as  reported  by  the  team  members 
served  as  the  dependent  variable,  while  for  the  elicited  factors,  the 
relative  importance  rating  given  that  factor  by  the  respondents  was  used. 

12.  Carter  et  al  .  (1973):  "A  Study  of  Critical  Factors  in 
Management  Information  Systems  for  U.S.  Air  Force" 

The  research  of  Carter  et  al .  is  presented  to  us  at  a  somewhat 
earlier  stage  than  the  other  studies  we  have  considered.  The  long  term 
goals  of  this  research  are  the  identification  of  those  factors  critical 
to  MIS  implementation  success,  the  development  of  instruments  to  measure 
the  status  of  these  factors,  and,  eventually,  the  development  of  a 
model  for  predicting  implementation  success.  The  phase  I  results 
presented  relate  primarily  to  the  first  of  these  three  goals.  An  initial 
set  of  interviews  with  systems  and  general  management  professionals  was 
used  to  generate  a  series  of  reconstructed  implementation  case  histories. 
The  researchers  "content  analyzed"  these  interviews  to  abstract  a  list 
of  critical  factors.  A  subset  of  this  list  was  tested  in  a  national 
survey,  in  which  respondents  were  asked  to  rank  the  relative  importance 
of  the  fourteen  factors  presented.  Thus,  at  this  stage  of  the  research, 
the  average  ranking  (in  terms  of  its  overall  importance)  received  by  a 
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factor  serves  as  the  dependent  variable.  We  should  note  that  this 
project  is  considerably  more  sophisticated,  both  in  terms  of  scope  and 
method,  than  much  of  the  other  factor  research;  the  full  project  involves 
several  recursive  steps  over  a  period  of  a  few  years. 

13.  Bean  et  al .  (1973):  "Structural  and  Behavioral  Correlates 
of  the  Implementation  of  Formal  OR/MS  Projects:  Success 
and  Failure  in  U.S.  Business  Organizations" 

This  is  the  most  ambitious  study  (at  least  in  terms  of  sample 
size)  included  in  our  sample.   It  was  aimed  primarily  at  the  structural 
and  organizational  factors  affecting  implementation.  One  hundred  and 
eight  companies  covering  12  industry  sectors  were  explored,  and  approx- 
imately 10-15  projects  in  each  company  were  included  in  the  sample. 
Factors  were  pre-selected,  and  were  measured  through  structured  inter- 
views with  the  head  of  the  OR/MS  activity  in  each  company.  Two  dependent 
variables  were  considered;  the  percentage  of  projects  worked  on  which 
were  successfully  implemented,  and  the  OR/MS  activity  manager's  overall 
assessment  of  his  group's  success  (an  index  composed  of  a  number  of  his 
responses).  One  particularly  important  finding  of  this  study  is  the 
effect  of  the  OR/MS  activity's  "stage  in  the  life  cycle"  on  both  which 
factors  are  important  and  the  overall  level  of  OR/MS  success. 

14.  Schultz  &  Slevin  (1973):  "Implementation  and  Organiza- 
tional Validity:  An  Empirical  Investigation" 

The  final  study  in  our  sample  is  a  report  of  Schultz  and 
Slevin's  efforts  to  develop  an  easily  usable  instrument  for  studying 
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implementation.  The  authors  contend  that  the  relative  lack  of  research 
on  implementation  is  due,  at  least  in  part,  to  a  lack  of  research  tools; 
hence  their  effort  to  develop  a  tool  that  would  be  generally  applicable 
for  such  research.  The  instrument  and  this  study  focus  on  individual 
attitudes  as  they  relate  to  a  particular  model  and  its  implementation. 
An  extensive  list  of  potential  variables  was  assembled  from  the  litera- 
ture, and  a  pilot  version  of  the  questionnaire  was  administered  to  a 
class  of  M.B.A.  students.  The  responses  were  factor  analyzed,  and  a 
modified  questionnaire,  based  on  these  extracted  factors  was  used  in  the 
field  test.  This  latter  test  involved  94  sales  and  marketing  personnel 
in  a  single  company,  responding  to  questions  concerning  a  specific  model 
under  development  for  their  department.  Questionnaire  items  were  of 
two  types  —  semantic  differential  concept  factors  and  specific  items  in 
a  Likert-type  format.  Five  of  these  latter  items,  dealing  with  expecta- 
tions for  the  model's  use  and  worth,  served  as  dependent  variables. 

This  section  has  reviewed  the  issues  of  setting,  methodology, 
and  dependent  variable  in  the  fourteen  studies  that  comprise  our  sample 
of  factor  research.  Table  I  provides  a  summary  of  this  material.  As  can 
be  seen,  the  studies  cover  a  wide  range.  Beyond  the  questions  of  setting 
and  method,  the  issue  of  focus  has  been  raised.  Some  studies  were 
concerned  with  the  factors  affecting  specific  implementations,  others 
focused  on  factors  influencing  OR/MS  group  effectiveness,  while  still 
others  examined  the  issues  of  growth  and  diffusion  of  the  OR/MS  approach. 
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All  of  these  are   legitimate  areas  for  implementation  research;  all  are 
representative  of  some  part  of  the  factor  research  that  has  been  conducted 
Thus,  all  were  included  in  this  review.   In  the  next  section  we  turn  to 
the  results  of  these  studies. 

3.  Results  of  the  Factor  Studies 

3.1.  A  structure  of  implementation  factors. 

The  fourteen  studies  we  have  reviewed  present  us  with  140 
separate,  identifiable  factors.  Considering  all  of  these  factors  on  an 
individual  basis  would  be  both  time  consuming  and  dull.  Thus,  we  need 
some  sort  of  structure  within  which  to  organize  these  factors.  In 
reviewing  the  literature,  and  in  attempting  to  think  about  the  various 
possible  influences  on  implementation  outcomes,  we  have  found  ten  major 
clusters  of  variables.  These  are: 

1.  Symptoms,  actions,  and  specific  behaviors  --  identifiable 

actions  taken  by  the  user,  designer,  or  management  which 
either  affect  or  are  in  some  way  symptomatic  of  the 
implementation  effort; 

2.  Internal  ecology  --  characteristics  of  the  organization, 

its  structure,  and  the  people  who  comprise  it; 

3.  Model  characteristics  --  characteristics  of  the  physical 

and  conceptual  solutions  to  the  problem; 

4.  Problem  characteristics  --  the  nature  of  the  problem  and 

its  criticality  to  the  organization; 
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5.  Extra-organizational  environment  --  characteristics  of  the 

external  environment  facing  the  organization; 

6.  Implementation  process  characteristics  --  methods  used  to 

move  through  the  stages  of  system  development,  cormuni- 
cation  patterns,  role  structures,  etc.; 

7.  Perceptions  --  perceptions  of  tasks,  goals,  and  the  impact 

of  the  proposed  solution  held  by  those  involved  in  or 
affected  by  the  implementation  effort; 

8.  Expressed  attitudes  --  attitudes  of  involved  and  effected 

personnel  towards  technology,  the  problem,  and  one 
another; 

9.  Underlying  attitudes  and  motives  --  more  basic  outlooks 

and  desires  of  involved  and  effected  personnel;  and 
10.  Organization  history  --  past  actions  of  the  organization 

which  can  impact  its  current  ability  to  deal  with 

innovation. 
Each  of  these  major  clusters  has  been  divided  into  sub-clusters  based 
on  the  particular  factors  explored  in  the  fourteen  studies  of  our  sample 
(see  Appendix  I).  We  make  no  claim  for  the  completeness  of  the  sub- 
clusters;  these  are  based  solely  on  the  140  identified  factors.  We  do 
suggest,  however,  that  the  ten  major  grou|)s  cover  the  range  of  potential 
factors  adequately. 


-22- 


3.2.  Summary  of  results. 

Appendix  I  presents  a  matrix  of  the  140  factors,  grouped  in 
accord  with  the  structure  we  have  presented  above,  showing  the  results 
of  the  fourteen  studies  we  have  considered.  Perhaps  the  most  striking 
characteristic  of  this  matrix  is  its  sparseness.  Fully  102  (73%)  of 
the  factors  are  reported  in  only  one  of  the  fourteen  studies.  An 
additional  23  (16%)  are  reported  in  two  studies,  and  12  (9%)  in  three. 
Of  the  remaining  three  factors,  two  --  "well  defined  measurable 
objectives"  and  "complexity  of  techniques  and  models"  --  are  considered 
in  four  of  the  research  efforts  reviewed,  and  the  final  one  -- 
"management  support"  --  sees  daylight  five  times.  Thus,  from  the  start 
we  must  realize  that  we  are  not  looking  at  issues  widely  tested  in  a 
variety  of  settings. 

Next,  we  note  the  concentration  of  both  factors  and  observa- 
tions in  the  first  few  of  our  ten  factor  groups.  Fully  50%  of  the 
factors  and  58%  of  the  factor  observations  fall  into  the  first  two 
classes  --  symptoms  and  actions,  and  characteristics  of  the  internal 
organizational  environment.  This  translates  to  28  of  the  38  (or  74%) 
factors  with  multiple  observations  falling  into  the  first  two  classes, 
with  87%  (33)  falling  in  the  first  four  of  ten  groups.  We  will  return 
to  this  issue  in  the  next  section  when  we  consider  more  carefully  the 
types  of  factors  that  have  been  studied. 

We  have  already  pointed  to  the  very  low  degree  of  overlap 
among  factors  considered  in  these  fourteen  studies.  The  importance  of 
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this  lack  of  comparability  is  emphasized  when  we  consider  those  factors 
where  multiple  measures  have  been  taken.   In  a  significant  proportion 
of  such  cases  there  is  a  lack  of  concordance  among  the  findings. 
Consider,  for  example,  the  factor  "well  defined,  measurable  objectives 
for  the  system"  (I. CIO).  Four  studies  included  this  factor;  in  three 
it  was  found  to  be  important,  while  in  the  fourth  (Dickson  &  Powers)  it 
did  not  relate  significantly  to  any  of  the  four  dependent  variables. 
Numerous  similar  examples  can  be  found  in  the  tables  of  Appendix  I. 

A  partial  explanation  for  this  finding  is  the  difference  in 
dependent  variables  among  the  studies.  This  can  be  seen  most  clearly 
in  the  results  of  the  Dickson  &  Powers  and  Bean  et  al .  studies.  In 
both  of  these  efforts  multiple  dependent  variables  were  employed  --  four 
in  the  former  and  two  in  the  latter  --  and  the  authors  report  the 
correlations  of  each  dependent  variable  with  the  factors.  Thirty 
percent  (12  out  of  40)  of  the  factors  reported  in  the  Bean  et  al.  study 
show  different  results  depending  on  which  dependent  variable  --  percent 
of  initiated  projects  actually  implemented  or  overall  rating  of  the  OR/MS 
group's  success  --  is  used.  Of  course,  the  differences  for  any  given 
factor  are  only  between  significance  in  one  direction  and  no  significant 
relationship.  However,  in  the  Dickson  &  Powers  study  the  picture  is 
somewhat  more  depressing;  no  factor  significantly  related  (at  the  10% 
level)  to  more  than  one  of  the  dependent  variables  affects  all  of  the 
dependent  variables  it  is  significantly  related  to  in  the  same  direction. 

While  the  differences  in  dependent  variables  employed  provide 
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a  fairly  convincing  explanation  for  the  apparent  contradictions  among 
certain  of  the  results,  there  are   other  potential  explanations  which  we 
should  explore.  One  such  possibility  is  a  lack  of  validity  in  the 
factor  measures;  i.e.,  the  'instruments'  used  do  not  measure  what  they 
purport  to.  The  results  reported  in  the  fourteen  studies  do  not  give 
us  much  opportunity  to  test  this  hypothesis,  but  there  is  one  indication 
that  this  might  be  the  caso.  Schultz  &  Slevin  use  a  combination  of 
semantic  differential  scali^s  and  Likert-type  questions  in  their  work, 
and  for  a  few  factors  theso  two  measures  overlap.  For  one,  "expected 
changes  in  the  communication  system  and  interpersonal  relations",  the 
results  differ  for  the  two  ways  of  measuring  the  factor.  While  this  is 
hardly  conclusive  proof  of  poor  instrument  validity  throughout  the 
factor  studies,  it  does  suggest  that  the  problem  may  exist  (particularly 
when  we  consider  that  many  studies  relies  on  the  researcher's  observa- 
tions for  'factor  measurement',  and  that  there  is  no  evidence  to  suggest 

* 
that  any  instrument  was  ever  used  more  than  once). 

The  third  potential  explanation  is  the  issue  of  contingency, 

or  lack  of  comparable  conditions.  Implicit  in  the  factor  approach  is 

the  assumption  that  factors  are  universal  --  management  involvement  is 

always  good,  or  the  OR  group's  selecting  the  projects  to  be  worked  on  is 

always  bad.  We  contend  that  this  is  an  unwarranted  assumption,  that  the 

importance  of  a  given  factor  will  vary  from  situation  to  situation. 


* 
Keen  (1974)  takes  a  closer  look  at  the  instruments  used  in  factor  studies 

and  the  question  of  their  validity. 
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Bean  et  al.  present  some  evidence  which  supports  this  position.  They 
analyze  their  data  both  in  aggregate  and  broken  down  to  control  for  the 
stage  in  the  life  cycle  of  the  organization's  OR/MS  activity,  and  find 
differences  in  factor  significance  across  the  stages.  This  is  only  one 
of  many  possible  contingencies,  but  it  points  up  the  need  to  identify 
the  relevant  contingencies  before  accepting  the  results  of  a  given  study. 

Finally,  in  considering  explanations  for  particular  results,  we 
should  not  ignore  the  possibility  of  spurious  correlation  or  non- 
correlation.  Turning  again  to  Bean  et  al.,  we  observe  a  most  unusual 
finding:  the  performance  of  post  implementation  evaluations  is  signifi- 
cantly positively  related  to  the  percent  of  initiated  projects  which  are 
implemented,  but  is  unrelated  to  the  overall  rating  of  the  OR/MS  group's 
success.   It  is  indeed  difficult  to  imagine  why  performing  post  project 
audits  should  affect  the  rate  at  which  projects  are  accepted.  At  best 
we  n)ight  believe  that  such  an  administrative  procedure  is  related  to 
other  organizational  practices  which  in  turn  im|iact  implementation  rate, 
but  accepting  this  result  as  it  stands  requires  just  too  great  a  leap 
of  faith. 

We  can  draw  certain  conclusions  from  our  brief  look  at  the 
factor  study  results.  Though  many  factors  have  been  studied  there  are 
few  general  guidelines  emerging  from  this  line  of  research.   In  part  this 
is  due  to  a  lack  of  replication  of  tests,  and  in  part  to  the  fact  that 
little  of  what  has  been  reported  prior  to  a  given  study  seems  to  find  its 
way  into  the  design  of  that  study.  This  is  further  compounded  by  the 
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use  of  different  instruments  and  dependent  variables  in  each  study,  and 
by  a  failure  to  identify  the  contingencies  that  are   important  in  any 
given  situation.  All  in  all,  we  can  judge  this  line  of  research  as 
having  contributed  less  than  it  might  have  so  far.  We  would  also  suggest 
that  the  factor  approach,  lacking  in  theory,  inter-study  comparability 
and  to  a  great  extent,  direction,  can  never  produce  more  than  fragmented 
findings  of  the  type  it  has  so  far.  Until  there  is  an  effort  to  build 
"maps"  which  can  integrate  results  by  focusing  on  key  contingencies,  we 
cannot  progress  in  this  line  of  research  (see  Keen,  1974,  for  further 
comments  on  this  issue). 

4.  A  Deeper  Look  at  the  Factor  Paradigm 
4.1.  Relationships  among  the  factors. 

In  the  previous  section  we  presented  ten  groups  into  which  we 
classed  the  140  factors.  Now  we  will  explore  these  classes  more  carefully, 
attempting  to  uncover  relationships  which  exist  among  the  different 
factor  classes.  We  can  think  about  factors  as  being  arrayed  along  a 
dimension  of  visibility  to  the  outside  observer.  At  one  extreme  would 
be  those  factors  which  are  totally  overt,  accessible  to  even  the  casual 
observer;  and,  at  the  other  extreme,  those  which  are  almost  completely 
hidden,  inaccessible  to  all  but  those  with  a  considerable  knowledge  of 
the  particular  organization,  its  operations,  and  its  background.  We 
suggest  that  the  ten  factor  classes  presented  earlier  form,  roughly,  such 
a  continuum,  and  can  be  divided  into  a  few  groups.  At  the  overt  extreme 
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are  the  first  two  classes  --  symptoms,  actions,  and  specific  behaviors, 
and  characteristics  of  the  internal  organizational  ecology;  the  factors 
in  these  two  groups  being  relatively  easy  to  observe  in  most  organiza- 
tions. At  the  opposite  extreme  are  categories  IX  and  X  --  underlying 
attitudes  and  motivations,  and  aspects  of  the  organization's  history. 
Clearly,  these  factors  are  largely  hidden  from  all  but  the  most 
persistent  and  resourceful  observers.  In  between  are  categories  III 
through  VIII.  These  might  be  divided  into  two  or  three  groups,  but 
it  is  not  critical  to  do  so  for  our  purposes.  We  have  arranged  them 
in  an  order  we  believe  approximates  the  degree  to  which  they  are  hidden 
from  the  observer  --  model  characteristics  being  most  overt  and 
expressed  attitudes  most  covert. 

Keeping  this  overt-hidden  dimension  in  mind,  lets  look  at  the 
factors  from  another  angle.  Implementation  is  a  process  of  change. 
As  a  process,  it  accepts  certain  inputs  and  produces  certain  outputs. 
The  critical  output  of  such  a  change  process  is  the  desired  change; 
in  the  case  of  MS  implementation,  the  model  or  system  and  the  requisite 
changes  in  behavior  to  operate  with  the  new  system.  The  inputs  to  the 
process  include  the  attitudes  and  actions  of  the  individuals  involved, 
their  perceptions,  the  problem  they  are  addressing  and  the  technology 
they  attack  it  with,  and  the  environment  in  which  they  are  operating. 
In  other  words,  many  of  the  factors  mentioned  in  the  factor  studies 
are  inputs  to  the  implementation  process.  Others,  particularly  those 
listed  under  classes  I  (symptoms,  actions,  and  specific  behaviors)  and 
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Table  II 


Distribution  of  Research  Effort  by  Factor  Class 
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By  research  effort  we  mean  the  inclusion  of  a  factor 
in  a  study,  or  to  be  more  presise,  the  reporting  of  a 
factor  in  the  write-up  of  a  study.  Thus,  the  numbers 
in  the  body  of  this  table  are  the  percentages  of  the 
factors  in  the  particular  study  (column)  falling  into 
that  class  (row).  The  final  column  presents  these  same 
percentage  for  the  fourteen  studies  taken  as  a  group, 
and  the  final  row  shows  the  proportion  of  factor  observa- 
tions attributable  to  each  study.  The  paranthesized 
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Table  II  (continued) 


numbers  in  the  final  row  and  column  represent,  respect- 
ively, the  number  of  factors  included  in  the  study, 
and  the  number  of  observations  (i.e.,  inclusions  in  a 
study)  of  the  factors  in  that  class. 


VI  (implementation  process  characteristics)  are  an  indication  of 
the  process  that  is  taking  place;  they  are  symptoms  of  the  on-going 
process.  We  should  note  that  neither  of  these  types  of  factors  truly 
represents  the  actual  process  of  change.  On  the  one  hand  we  have  the 
'fuel'  for  the  process;  on  the  other,  certain  side  effects  of  its 
occurring.  But,  the  former  group  does  not  represent  the  cause  of 
change,  nor  the  latter  its  output. 

Returning  now  to  our  overt-hidden  dimension,  we  can  see  how 
the  factors  fit  together.  At  the  overt  extreme  of  the  spectrum  we 
have  the  bulk  of  the  symptoms  of  change  plus  one  type  of  input  to 
the  process.  The  remainder  of  the  inputs  are  arrayed  along  the  scale 
towards  the  hidden  end. 

When  we  look  again  at  the  fourteen  studies  we  find  a 
disproportionate  amount  of  research  effort  going  into  the  overt  end  of 
the  scale  (Table  II).  The  first  two  factor  classes  represent  a  total 
of  58%  of  the  factor  observations  (23%  and  35%  respectively).  None  of 
the  remaining  eight  classes  received  more  than  8%  of  the  total  factor 
observations.  Looking  at  the  studies  individually  we  see  that  all 
fourteen  consider  some  factors  from  one  of  the  two  first  groups.  In 
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addition,  in  all  but  three  of  the  studies  the  class  receiving  the  most 
attention  was  one  of  the  first  two.  No  other  factor  class  comes  near 
these  first  two  in  terms  of  attention  received. 

We  have  no  a  priori  reason  to  believe  that  overt  behaviors 
or  internal  organizational  ecology  are  more  important  than  other 
factors  in  determining  implementation  outcomes.  In  fact,  a  case  can 
be  made  which  suggests  just  the  opposite;  attitudes,  perceptions, 
motives,  and  organizational  history  may  well  be  the  underlying 
causes  of  some  of  the  more  overt  factors,  specific  actions  and  behaviors 
or  the  structuring  of  the  internal  organization,  for  example.  Why, 
then,  do  we  find  the  preponderance  of  research  effort  being  directed 
towards  the  most  overt  factors?  Perhaps  the  most  plausible  explanation 
is  that  suggested  by  Keen.  The  more  overt  the  factor,  the  easier  it 
is  for  a  researcher  to  measure  it.  Having  little  or  no  theory  to  guide 
them  in  their  selection  of  factors,  researchers  have  chosen  those 
factors  which  were  most  readily  accessible. 

4.2.  Further  work  with  the  factor  paradigm? 

There  has  been  a  trend  very  recently  in  some  circles  to  move 
away  from  the  factor  approach  to  implementation  research  (Gibson,  1973, 
Lucas  &  Plimpton,  1972,  Sorensen  &  Zand,  1973,  and  Urban,  1974,  are  all 
examples).  The  reason  for  this  movement  is  a  growing  recognition  of 
some  serious  shortcomings  of  the  factor  paradigm.  Most  important  among 
these  failings  are  the  absence  of  any  theoretical  base  supporting  most 
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factor  studies,  a  failure  to  look  at  implementation  as  a  process 
extending  over  time,  and  a  tendency  to  focus  on  a  few  variables  one 
at  a  time,  thus  ignoring  other  important  variables  and  interactions 
among  variables. 

We  will  turn  now  to  one  recent  study  which  was  undertaken  to 
meet  the  objections  cited  above  (Miller,  1974).  This  study  is  an 
extension  of  the  work  of  Gibson  and  his  colleagues,  and  took  place  at 
the  same  field  site.  Thus,  the  author  had  a  considerable  amount  of 
backgroung  knowledge,  both  substantive  and  methodological,  to  draw  upon. 
Miller  suggests  that  in  order  to  understand  implementation  it  will  be 
necessary  "to  conduct  longitudinal  studies  of  the  implementation 
process  in  its  manifold  complexity  as  it  unfolds  and  develops  over 
time"  (p.  13).  His  aim  is  to  understand  the  behaviors  of  managers  and 
model  builders  which  together  bring  about  the  outcomes  of  an  implement- 
ation effort.  To  this  end,  he  adopts  a  simple  model  for  viewing  such 
behavior.  The  model  proposes  that  three  sub-systems  (or  variable 
classes)  determine  behavior  --  the  individual,  the  social,  and  the 
organizational  sub-systems.  For  each  of  these  sub-systems  Miller 
defines  certain  variables  and  measures  assumed  to  be  the  right  ones 
for  consideration  in  an  attempt  to  understand  implementation.  Over  a 
relatively  long  period  the  researcher  then  observed  an  implementation 
effort  (the  N.E.  bank  translator  model  mentioned  in  our  discussion  of 
Gibson's  work)  and  collected  a  considerable  quantity  of  data,  including 
quantitative  measures  of  the  sub-system  indicators  he  had  defined. 
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And,  in  qeneral ,  he  found  that  these  variables  had  some  impact  on  the 
implementation  process. 

Miller  undertook  this  research  effort  in  answer  to  certain 
shortcomings  he  found  in  the  factor  approach,  and  in  certain  areas  he 
has  successfully  addressed  problems  which  characterize  previous  factor 
studies.  He  recognizes  implementation  as  a  process  of  change.  Hence, 
he  has  chosen  a  dependent  variable  more  appropriate  to  such  an  effort 
than  the  simple  definitions  of  success  frequently  used  in  factor 
studies;  and,  he  has  looked  at  the  change  effort  over  a  sufficiently 
long  period  of  tine  to  get  some  real  understanding  of  what  is  going 
on.  However,  in  many  ways  his  work  is  really  a  factor  study,  and  he' 
falls  into  some  of  the  traps  of  previous  factor  researchers. 

By  carefully  examining  the  problems  which  Miller's  work  raises, 
we  can  begin  to  understand  why  the  factor  paradigm  is  an  inadequate 
model  for  implementation  research.  First,  consider  the  variables 
studied  in  this  effort.  Table  III  presents  these  variables,  where  they 
fit  in  the  structure  of  factors  presented  earlier,  and  an  indication 
of  whether  or  not  they  were  included  in  one  (or  more)  of  the  fourteen 
studies  we  have  reviewed.  Three  points  are  immeaiately  apparent  when 
we  look  at  the  table.  First,  Miller's  factors  fall  largely  (five  out 
of  seven)  in  a  single  factor  class,  the  internal  organizational 
environment.  Second,  coverage  is  strongly  biased  towards  the  more 
overt  factors,  with  five  factors  in  class  II  and  one  in  class  I.  And, 
third,  the  factors  studied  are  by  and  large  new  ones,  not  covered  in 
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Table   III 


Factors   in  Miller's  Study  of  Implementation 


Factor 


1.  cognitive  style  of  users  & 
model  builders 

2.  builder/user  personality 
style  match 

3.  builder/user  status  or 
rank  differences 

4.  belief  or  ideological  systems 
of  builder  &  user 

5.  builder  &  user  task  environ- 
ments --  explicitness  of  task 
definitions,  constraints 


Cluster, 
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stud 
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II. F 
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II. F 

0 

IX  &  IX 
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er;  1-user 

imposed,  &  nature  of  tasks 

II 

0 

6. 

builder  &  user  reward  struc- 

tures 

II 

0 

7. 

supervisory  support  for 

assigned  roles 

I. A 

covered  in  part 
by  I.A.2&3 

any  of  the  fourteen  studies;  two  of  the  seven  factors  being  partially 
addressed  in  other  studies,  and  only  one  factor  that  truly  overlaps 
those  in  the  previous  studies. 

What,  we  might  ask,  are  the  implications  of  these  results? 
Let  us  start  with  the  third,  the  introduction  of  new  factors.  We 
observed  earlier  that  a  wery   large  portion  of  the  140  factors  in 
Appendix  I  were  addressed  in  only  one  study.  Now  we  add  one  more  study 
including  seven  factors  and  our  total  jumps  to  146.  The  problem  this 
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suggests  is  that  the  number  of  potentially  important  factors  is  virtually 
uncountable.  So  many  variables  could  affect  an  implementation  effort 
that  we  could  go  on  forever  adding  new  factors  to  our  list. 

Turning  to  the  issue  of  coverage  of  the  range  of  factors,  we 
can  see  that  Miller  fell  into  the  trap  of  most  previous  factor  research- 
ers. He  opted  to  measure  a  certain  subset  of  factors,  and  these  were 
concentrated  at  the  overt  end  of  the  factor  spectrum.  True,  he  found 
these  seven  variables  to  be  significant,  but  he  can  say  nothing  about 
the  hundreds  of  factors  he  did  not  test.  This,  we  believe,  is  one  of 
the  major  weaknesses  of  the  factor  paradigm;  you  can  only  get  out  what 
you  put  in.  And,  this  is  true  whether  the  factors  to  be  measured  are 
picked  at  random,  selected  on  the  basis  of  ease  of  measurement,  or 
based  on  some  theory  of  behavior.  The  initial  selection  determines 
the  maximum  that  you  can  learn.  We  will  discuss  this  point  further 
in  the  next,  and  final,  section  of  this  paper. 

5.  Summary  of  Problems  with  the  Factor  Model 

Factor  studies  of  implementation  have  given  us  some  useful 
knowledge  about  implementation.  They  have  shown  that  there  are  a  myriad 
of  factors  which  can,  and  at  least  sometimes,  do  affect  the  progress 
of  an  implementation  effort.  However,  we  contend  that  this  approach 
has  a  number  of  flaws  serious  enough  to  cause  us  to  doubt  its  further 
usefulness  as  a  paradigm  for  research  on  implementation. 

The  first  of  these  problems  stems  from  the  almost  limitless 
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number  of  potentially  relevant  factors.  The  researcher,  having  only 
a  limited  amount  of  time,  must  of  necessity  limit  his  investigation  to 
a  small  subset  of  these  factors.   In  the  absence  of  any  well  defined 
theory  of  implementation,  the  selection  of  factors  for  consideration  is 
left  largely  to  the  individual  researcher's  biases;  and,  as  we  have  seen, 
such  selection  is  not  likely  to  provide  a  representative  sample  of 
factors.  We  have  also  noted  that  the  same  result  can  obtain  if  we  move 
to  the  other  extreme.  Tightly  defined  theories  of  implementation 
behavior  are  likely  to  focus  on  only  a  few  factors.  Thus,  using  such 
theories  as  guides  to  the  selection  of  factors  for  study  yields  the 
same  result  as  using  no  theory  at  all;  careful  attention  to  a  small 
number  of  factors  at  the  expense  of  ignoring  other,  likely  equally  or 
more  important,  factors.  As  we  stated  earlier,  you  can  only  find  out 
what  you  go  in  looking  for. 

The  second  major  defficiency  of  the  factor  model  as  a  research 
vehicle  is  its  inherently  static  view  of  the  world.  Typically,  factors 
are  measured  at  a  single  point  in  time  and  the  results  are  assumed  to 
capture  the  essence  of  that  implementation  effort.   If  we  review  the 
factors  listed  in  Appendix  I,  we  find  that  all  of  them  deal  with  either 
inputs  to  the  implementation  process  or  side  effects  of  the  process. 
None  are   truly  concerned  with  the  dynamics  of  the  on-going  process. 
Yet,  implementation  is  a  process,  taking  place  over  a  considerable  span 
of  time,  and  any  meaningful  understanding  of  implementation  must  stem 
from  a  recognition  of  its  inherently  dynamic  nature. 
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Our  third  complaint  concerns  the  implicit  assumption  of  many 
factor  researchers  that  factors  are  absolutes  --  a  given  condition  is 
always  good  or  always  bad.  A  more  realistic  view  is  one  which  considers 
contingencies  or  factor  interactions.  The  effect  of  a  factor  in  a  given 
situation  depends  greatly  on  the  other  factors  present  in  that  situation. 
There  is  no  inherent  reason  (other  than  the  complexity  of  analysis  and 
the  need  for  larger  quantities  of  data)  why  factor  studies  cannot  consider 
such  interactions;  few  researchers,  however,  have  done  so. 

The  fourth,  and  final,  fault  we  will  raise  is  the  failure  to 
focus  on  the  management  of  the  implementation  process.  The  concern  has 
been  with  measuring,  classifying  conditions  as  favorable  or  unfavorable 
to  implementation  outcomes.  This  orientation  is  understandable  given 
the  static  world  view  embodied  in  this  paradigm  and  the  nature  of  the 
factors  that  have  been  considered.  Practitioners,  however,  are  more 
interested  in  managing  the  process,  making  an  implementation  effort 
successful,  than  in  being  able  to  explain  why  it  failed. 

In  sum,  we  can  state  four  necessary  characteristics  of  an 
improved  implementation  research  methodology.  First,  it  should  be 
grounded  in  a  sort  of  "mid-range"  theory;  going  neither  to  the  extreme 
of  no  theory  at  all,  nor  to  the  that  of  a  theory  so  confining  that  much 
important  evidence  is  ignored.  Next,  it  should  focus  on  dynamics,  the 
process  aspects  of  implementation.  Third,  it  must  recognize  the  exist- 
ence of  contingencies  and  interactions  among  factors,  and  it  should 
aim  towards  developing  maps  of  the  major  and  critical  contingencies 
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likely  to  arise  in  given  situations.  Finally,  if  it  is  to  be  of  any 
real  use,  it  must  be  oriented  towards  the  management  of  the  process, 
and  must  include  appropriate  control  points  for  managers'  and 
practitioners'  use. 


* 
See  Keen  (1974)  for  a  more  thorough  discussion  of  the  desiderata  of  a 


methodology  for  implementation  research. 
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Appendix  I:   Findings  of  the  Factor  Studies 


Key 


Since  the  studies  vary  in  what  they  report,  it  is  not  possible 

to  adopt  a  uniform  coding  for  all  of  the  study  results.  Thus, 

each  of  the  studies  is  marked  in  the  following  table  in  the 
manner  which  fits  it  most  logically. 


Where  applicable  factors  are  coded  as  follows: 

+  --  significant  (by  whatever  test  was  used  in  the  particular 
study)  positive  effect  of  the  factor  on  the  dependent 
variable 

-  --  significant  negative  effect  of  the  factor  on  the 
dependent  variable 

0  --  no  significant  effect  of  the  factor  on  the  dependent 
variable  found 


Exceptions  to  the  above  scheme  are: 

1.  Rubenstein  et  al .      --     a    '•^'    indicate?    those  variables 

found  to  be  favorable  to  implementation  success. 

2.  Harvey  --  a  ' *^ '  indicates  those  faclors  suggested  to  be 

important. 

3.  Dickson  &  Powers  --  factors  marked  with  the  direction  (+  or  -) 

for  each  of  the  dependent  variables  they  were  found  to 
affect  significantly  (T  --  time,  B  --  budget,  U  --  user 
satisfaction,  C  --  computer  operations),  or  with  a  '0'  if 
no  dependent  variable  was  found  to  be  significantly 
affected. 

4.  Vertinsky  --  a  '  »^  '  indicates  those  factors  found  to  be 

important. 

5.  Lucas  --  a  '  »^^  '  indicates  that  the  factor  entered  into  at 

least  one  of  the  'prediction  eq'iations'   --  N_.J^  all 
factors  tested  entered  into  at  least  one  equation. 
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6.  Manley  --  five  external  factors  marked  according  to  their 

relative  importance  as  determined  by  the  study  (1  =  most 
important);  motivational  factors  (attitude  orientations 
towards  the  external  factors)  marked  with  a  '  »x  ' . 

7.  Gibson  --  a  ' «^ '  indicates  those  variables  emerging  as 

important  in  the  case  studied. 

8.  Smith  et  al .  --  quantitatively  measured  factors  indicated 

with  the  standard  +,-,  or  0;  qualitatively  assessed  factors 
marked  for  their  relative  (to  one  another)  importance  as 
determined  in  the  study  (H  =  high,  M  =  medium,  L  =  low). 

9.  Carter  et  al .  --  ranking  of  the  relative  importance  assigned 

the  factors  included  in  the  nationwide  study  (H,  M,  or  L). 

10.  Bean  et  al .  --  marked  as  +,-,  or  0  for  each  dependent  variable 

separately:   implementation  rate/overall  success. 

11.  Schultz  &  Slevin  --  standard  +,-,  or  0  marking  except  in 

cases  where  the  two  ways  of  measuring  a  factor  yielded 

different  results;  then  marked  as  semantic  differential 

f inding/Likert  factor  finding. 

N.B.  factors  included  in  the  table  are  the  concepts  used 
in  the  sematic  differential  instrument  and  the 
derived  factors  composed  of  aggregates  of  items  from 
the  Likert-type  instrument. 
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E(cond) DISCRETION  GIVEN  TO  OR  GROUP 
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